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Abstract 


This  paper  proposes  the  use  of  a  new  Systems  Readiness  Level  (SRL)  scale  for 
managing  system  development  and  for  making  effective  and  efficient  decisions  during  the 
defense  acquisition  process.  This  scale  incorporates  both  the  current  Technology 
Readiness  Level  (TRL)  of  the  Department  of  Defense  (DoD)  and  the  concept  of  an 
Integration  Readiness  Level  (IRL)  developed  by  Stevens  Institute  of  Technology.  The  paper 
describes  the  foundations  for  the  SRL  and  how  it  is  formulated;  it  also  demonstrates  the 
SRL’s  application  within  the  defense  acquisition  process  using  a  sample  case  with  notional 
readiness  values. 
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1.  Introduction 


In  1999,  the  United  States  (US)  General  Accounting  Office  (GAO)1  stated  that 
there  were  few  metrics  used  within  the  US  Department  of  Defense  (DoD)  to  gauge 
the  impact  of  investments  or  the  effectiveness  of  processes  used  to  develop  and 
transition  technologies.  It  asserted  that  additional  metrics  in  technology  transition 
were  needed  (GAO,  1999).  In  2002,  in  a  testimony  before  the  Subcommittee  on 
Readiness  and  Management  Support,  Committee  on  Armed  Services  of  the  US 
Senate,  the  GAO  further  explained  DoD  challenges  in  implementing  best  practices;  it 
suggested  the  DoD  needed  to  enable  success  through  the  demonstration  of  value 
and  the  credibility  of  new  processes  through  the  use  of  metrics  (GAO,  2002). 

To  address  these  compounding  challenges,  in  1999,  the  DoD  began 
implementing  the  Technology  Readiness  Level  (TRL)  as  a  metric  to  assess  the 
maturity  of  a  program’s  technologies  before  its  system  development  begins  (DoD, 
2005a;  2005b).  Additionally,  the  DoD  made  constructive  changes  to  its  approaches 
to  acquisition  that  would  address  these  issues  by  2001 :  (1 )  assuring  a  weapon 
systems’  technologies  are  demonstrated  to  a  high  level  of  maturity  before  beginning 
its  program  and  (2)  using  an  evolutionary  or  phased  approach  to  developing  such 
systems  (GAO,  2002). 

Even  with  the  implementation  of  new  processes  and  practices  within  DoD 
acquisition,  the  challenges  are  still  significant  (e.g.,  over  the  next  five  years,  the  DoD 
plans  to  invest  an  estimated  $900  billion  to  develop  and  procure  weapons  systems 
at  a  pace  that  far  exceeds  the  availability  of  resources  (GAO,  2008)). 

Consequently,  despite  the  utility  and  value  of  the  TRL  as  a  metric  for 
determining  technology  maturity  before  transitioning  into  a  system,  we  contend  that 
TRLs  were  not  intended  to  address  systems  integration  nor  to  indicate  that  the 


1  This  agency  became  the  US  Government  Accountability  Office  on  July  7,  2004. 
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technology  will  result  in  successful  development  of  a  system  (Gove,  2007; 
Mandelbaum,  2007;  2008).  As  Baines  (2004)  describes,  “the  wrong  technology,  or 
even  the  right  technology  poorly  implemented,  can  be  disastrous”  (p.  447. 

Therefore,  in  this  paper  we  will  build  upon  a  concept  originally  proposed  by  Sauser, 
Verma,  Ramirez-Marquez  and  Gove  (2006)  for  the  development  of  a  System 
Readiness  Level  (SRL)  scale  that  incorporates  the  maturity  level  of  the  critical 
components  and  the  interoperability  of  the  entire  system.  A  fundamental  argument 
to  this  approach  is  that  the  metrics  for  the  coupling  and  maturation  of  multiple 
technologies  and  systems  have  been  shown  to  be  unresolved  issues  of  strategic 
relevance  (Nambisan,  2002;  Watts  &  Porter,  2003).  In  addition,  component-level 
considerations  relating  to  integration,  interoperability,  and  sustainment  become 
equally  or  more  important  from  a  systems  perspective  during  acquisition  (Sandborn, 
Herald,  Houston  &  Singh,  2003). 

The  SRL  we  will  describe  and  demonstrate  is  a  function  and  scale  that 
incorporates  the  current  TRL  scale  along  with  a  scale  of  integration.  The 
combination  for  utilization  of  the  SRL  we  contend  aids  in  making  strategic  decisions 
during  defense  acquisition.  The  resultant  SRL  scale  can  provide  an  assessment  of 
overall  system  development  and  can  identify  potential  areas  that  require  further  work 
to  facilitate  prioritization.  This  new  SRL  scale  of  system  maturity  can  be  used  with 
decision-making  tools  for  the  potential  acquisition  of  systems — which  involve  the 
dependency  and  interplay  among  performance,  availability  (reliability, 
maintainability,  and  supportability),  process  efficiency  (system  operations, 
maintenance,  and  logistics  support),  and  system  lifecycle  cost. 
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2.  Theoretical  Foundation 


In  program  management,  resources  are  frequently  allocated  with  the  purpose 
of  executing  tasks  to  maintain  schedule  and  budget.  This  can  lead  to  an 
assignment-type  program  scheduling  problem  (Salewski,  Schirmer  &  Drexl,  1997) 
when  the  ultimate  objective  of  any  program  is  to  realize  a  product  (or  system)  to 
satisfy  a  customer.  A  fundamental  challenge  to  resolving  this  problem  is  that  when 
attempting  to  meet  the  emergent  needs  of  the  warfighter,  program  managers  (PMs) 
will  often  continue  development  of  a  system  through  the  acquisition  lifecycle — while 
they  coordinate  the  design  activities  with  preliminary,  ambiguous,  or  subjective 
information  (Pich,  Loch  &  De  Meyer,  2002).  The  balance  between  customer  needs 
(e.g.,  warfighter)  and  design  activities  creates  a  tension  between  the  overview 
required  by  the  program  manager  and  the  detail  that  is  the  focus  of  the  system 
developers  (de  Haes,  2006).  To  find  a  concession,  organizations  have  relied  on 
subjective  assessment  techniques  for  developing  the  program  overview,  which  then 
becomes  the  basis  for  making  strategic  acquisition  decisions.  However,  these 
subjective  assessments  are  human-intensive,  error-prone,  and  inadequate  for  the 
desired  management  controls;  such  controls  should  be  based  on  system  attributes 
that  can  be  quantitatively  measured  using  system  metrics  (Yacoub  &  Ammar,  2002). 
The  tension  between  subjectivity  and  detail  is  rationalized  through  prescriptive 
techniques — which  allow  people  to  make  better  decisions  by  using  normative 
models,  but  with  knowledge  of  the  limitations  and  descriptive  realities  of  human 
judgment  (Smith  &  Winterfeldt,  2004). 

Within  agencies  of  the  US  government,  the  prescriptive  tool  and  soft  metric  of 
the  TRL  has  been  used  as  an  assessment  of  the  maturity  of  evolving  technologies 
prior  to  incorporating  them  into  a  system  or  sub-system.  The  original  TRL  was  a  bi¬ 
product  of  the  National  Aeronautics  and  Space  Administration’s  (NASA)  post-Apollo 
era  as  ontology  for  contracting  support  (Sadin,  Povinelli  &  Rosen,  1989).  In  the  last 
nine  years,  other  government  agencies  and  contractors  have  adopted  the  TRL  scale 
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with  specific  variations  to  satisfy  their  needs  (e.g.,  the  Department  of  Defense  (DoD), 
the  Department  of  Energy  (DoE),  the  National  Air  and  Space  Intelligence  Center). 

There  have  been  many  attempts  to  identify  alternative  readiness/maturity 
levels  that  will  complement  the  TRL,  such  as  Design  Readiness  Level, 

Manufacturing  Readiness  Level,  Software  Readiness  Level,  Operational  Readiness 
Level,  Human  Readiness  Level,  Habitation  Readiness  Level  and  Capability 
Readiness  Levels  (Bilbro,  2007;  Connelly,  Daues,  Howard  &Toups,  2006;  Cundiff, 
2003).  Unfortunately,  each  has  faltered  in  addressing  the  core  issue  with  the  TRL 
as  identified  in  recent  literature;  thus,  the  legacy  constraints  with  the  TRL’s 
abstraction  have  remained.  These  constraints  are:  (1)  the  inability  to  represent 
integration  between  technologies,  (2)  an  uncertainty  in  the  maturation  of 
technologies,  and  (3)  an  inability  to  compare  the  impact  of  alternative  TRLs  on  the 
system  as  a  whole  (Cundiff,  2003;  Dowling  &  Pardoe,  2005;  Mankins,  2002;  Meystel, 
Albus,  Messina  &  Leedom,  2003;  Moorehouse,  2001;  Shishko,  Ebbeler  &  Fox,  2003; 
Smith,  2005;  Valerdi  &  Kohl,  2004). 

Based  on  these  fundamental  conjectures,  a  more  comprehensive  set  of 
concerns  becomes  relevant  when  the  TRL  is  amplified  from  the  level  of  an  individual 
technology  to  a  system  context  that  involves  the  interplay  of  multiple  technologies. 
For  example,  in  NASA’s  Mars  Climate  Orbiter,  the  failure  of  two — independently 
evaluated — technologies  to  use  the  same  units  (i.e.,  Metric  versus  English) 
contributed  to  the  loss  of  the  spacecraft.  While  testing  is  absolutely  necessary,  it  is 
not  always  capable  of  catching  the  many  small  errors  that  can  occur  when  two 
different  components  of  software  and/or  hardware  exchange  data  in  a  raw  format.  If 
the  integration  of  two  pieces  of  technology  followed  some  sort  of  maturation  process, 
just  as  the  technology  itself  does,  this  would  provide  an  assessment  of  integration 
readiness  and  a  direction  for  improving  maturity  from  a  systems  context  during  the 
development  process.  Not  withstanding  the  previously  identified  limitations  of  the 
TRL,  any  metric,  as  described  by  Dowling  and  Pardoe  (2005),  should  not  lose  sight 
of  what  makes  it  effective  and  efficient  in  an  organization: 
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1 .  The  way  the  value  is  used  should  be  clear. 

2.  The  data  to  be  collected  for  the  metric  should  be  easily  understood  and 
easy  to  collect. 

3.  The  method  of  deriving  the  value  from  the  data  should  be  clear  and  as 
simple  as  possible. 

4.  Those  for  whom  the  use  of  the  metric  implies  additional  cost  should 
see  as  much  direct  benefit  as  possible  (i.e.,  collecting  the  data  should 
not  cost  more  than  its  value  to  the  decision  process). 
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3.  Development  of  a  System  Readiness  Level 


In  theory,  technology  and  system  development  follow  similar  evolution  (or 
maturation)  paths;  a  technology  is  inserted  into  a  system  (e.g.,  evolutionary 
acquisition)  based  on  its  maturity,  functionality  and  environmental  readiness  and 
ability  to  interoperate  with  the  intended  system.  However,  many  of  the  factors  that 
may  determine  the  successful  deployment  of  a  system  into  its  operational 
environment  are  not  always  effectively  implemented  during  the  developmental 
lifecycle  (Parsons,  2006).  Fundamentally,  any  system  under  development  is 
composed  of  core  technology  components  and  their  linkages  in  accordance  with  the 
proposed  architecture.  Henderson  and  Clark  (1990)  showed  that  the  distinction 
between  the  relationships  of  the  components  and  the  system  architecture  requires 
two  types  of  knowledge:  component  knowledge  and  architectural  knowledge  (i.e., 
knowledge  on  how  the  components  are  integrated).  These  researchers  emphasized 
that  systems  often  fail  because  attention  is  given  to  the  technology  while  knowledge 
of  the  linkages/integrations  is  overlooked.  They  explain  that  improper  attention  to 
the  linkages/integrations  has  an  impact  on  the  systems’  technical  evolution, 
organizational  experience,  recurrent  task,  and  technical  knowledge  as  they  relate  to 
the  component  linkages.  It  also  influences  the  product  architecture,  communication 
channels,  and  problem  solving  strategies.  Therefore,  while  the  TRL  provides  the 
metric  for  describing  component  knowledge,  based  on  Henderson  and  Clark,  one 
would  still  be  interested  in  a  metric  that  provides  a  description  of  architectural 
knowledge  or  integration.  In  addition,  using  modeling  and  simulation,  Ford  and 
Dillard  (2008)  were  able  to  demonstrate  the  inherent  value  of  integration  to  the 
success  of  evolutionary  acquisition.  They  were  able  to  demonstrate  the  relative 
impact  of  making  integrations  decisions  late  in  the  acquisition  lifecycle. 

While  there  have  been  some  efforts  to  develop  metrics  that  can  be  used  to 
evaluate  integration  (e.g.,  DoD,  1998,  March  30;  Mankins,  2002;  Fang,  Hu  &  Han, 
2004;  Nilsson,  Nordhagen  &  Oftedal,  1990),  there  is  a  need  for  a  metric  that  can  be 
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used  with  the  TRL  to  effectively  determine  a  system  maturity.  This  paper  addresses 
this  need  by  developing  a  system  maturity  scale  that  incorporates  the  TRL  and  a 
metric  of  integration  maturity,  which  is  described  below. 

3.1.  Integration  Readiness  Level 

The  application  of  ontology  metrics  to  support  integration  has  been 
extensively  used  in  the  computer  industry  to  define  the  coupling  of  components 
(Orme,  Yao  &  Etzkorn  2006;  2007),  but  a  common  ontological  approach  to 
technology  integration  for  system  development  has  been  far  less  developed.  One  of 
the  first  attempts  to  address  this  was  conducted  by  Mankins  (2002)  when  he 
proposed  an  Integrated  Technology  Analysis  Methodology  to  estimate  an  Integrated 
Technology  Index  (ITI).  The  ITI  was  then  used  for  a  comparative  ranking  of 
competing  advanced  systems.  The  study  brought  to  the  forefront  the  difficulty  of 
progressing  through  the  TRL  scale  and  choosing  between  competing  alternative 
technologies.  It  did  not  adequately  address  the  integration  aspects  of  systems 
development.  Based  on  concerns  for  successful  insertion  of  technologies  into  a 
system,  the  Ministry  of  Defence  in  the  United  Kingdom  developed  a  Technology 
Insertion  Metric  that  includes,  among  other  things,  an  Integration  Maturity  Level 
(Dowling  &  Pardoe,  2005).  Building  upon  these  efforts,  Gove  (2007)  and  Gove, 
Sauser  and  Ramirez-Marquez  (2007)  performed  a  review  of  aerospace  and 
defense-related  literature  to  identify  the  requirements  for  developing  a  7-level 
integration  metric  that  they  called  Integration  Readiness  Level  (IRL).  These  factors 
led  to  the  definition  of  the  requirements  for  an  integration  metric,  which  are  to: 

Provide  an  integration-specific  metric,  to  determine  the  integration 
maturity  between  two  or  more  configuration  items,  components,  and/or 
subsystems. 

Provide  a  means  to  reduce  the  uncertainty  involved  in  maturing  and 
integrating  a  technology  into  a  system. 

Provide  the  ability  to  meet  system  requirements  during  the  integration 
assessment  so  as  to  reduce  the  integration  of  obsolete  technology 
over  less  mature  technology. 
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1. 

2. 

3. 


4.  Provide  a  common  platform  for  both  new  system  development  and 
technology  insertion  maturity  assessment. 

Using  these  requirements,  Gove  et  al.  (2007)  assessed  Mankin’s  Integrated 
Technology  Index  (  2002),  Nilsson  et  al.’s  integration  metric  (1990),  Fang  et  al.’s 
Interoperability  Assessment  Model  (2004),  and  their  7-level  IRL  (Sauser  et  al., 

2006).  While  none  of  these  methods  met  all  the  stated  requirements,  the  analysis 
yielded  a  modified  9-level  IRL  which  did.  The  resulting  IRL  is  a  systematic  analysis 
of  the  interfacing  of  compatible  interactions  for  various  technologies  and  the 
consistent  comparison  of  the  maturity  between  integration  points  (i.e.,  TRLs)  and  is 
described  in  Table  1 . 

Gove  et  al.  (2007)  also  evaluated  these  integration  maturity  metrics  with 
multiple  system  case  studies  (i.e.,  Mars  Climate  Orbiter,  Ariane  5,  two  Hubble  Space 
Telescope  cases)  to  determine  how  effective  they  would  be  in  recognizing 
integration  risks  in  development.  The  case  study  analysis  showed  that  the  existing 
approaches  to  integration  metrics  would  not  have  identified  the  root  cause  of  the 
development  risks.  Application  of  the  IRL  approach,  however,  was  shown  to  have 
highlighted  low  levels  of  integration  maturity  and  identified  specific  areas  of 
development  needing  further  management  and  engineering  attention. 

Consequently,  we  use  this  IRL  in  the  development  of  the  SRL. 
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Table  1.  Integration  Readiness  Levels 

(Gove,  2007;  Gove  et  al.,  2007) 


IRL 

Definition 

Description 

9 

Integration  is  Mission  Proven 
through  successful  mission 
operations. 

IRL  9  represents  the  integrated  technologies  being  used  in  the 
system  environment  successfully.  In  order  for  a  technology  to 
move  to  the  TRL  9,  it  must  first  be  integrated  into  the  system  and 
then  proven  in  the  relevant  environment;  thus,  progressing  IRL  to 

9  also  implies  maturing  the  component  technology  to  the  TRL  9. 

8 

Actual  integration  completed  and 
Mission  Qualified  through  test  and 
demonstration  in  the  system 
environment. 

IRL  8  represents  not  only  the  integration-meeting  requirements, 
but  also  a  system-level  demonstration  in  the  relevant 
environment.  This  will  reveal  any  unknown  bugs/defects  that 
could  not  be  discovered  until  the  interaction  of  the  two  integrating 
technologies  was  observed  in  the  system  environment. 

7 

The  integration  of  technologies  has 
been  Verified  and  Validated  with 
sufficient  detail  to  be  actionable. 

IRL  7  represents  a  significant  step  beyond  IRL  6;  the  integration 
has  to  work  from  a  technical  perspective,  but  also  from  a 
requirements  perspective.  IRL  7  represents  the  integration 
meeting  requirements  such  as  performance,  throughput,  and 
reliability. 

6 

The  integrating  technologies  can 
Accept,  Translate,  and  Structure 
Information  for  its  intended 
application. 

IRL  6  is  the  highest  technical  level  to  be  achieved;  it  includes  the 
ability  to  not  only  control  integration,  but  to  specify  what 
information  to  exchange,  to  label  units  of  measure  to  specify 
what  the  information  is,  and  the  ability  to  translate  from  a  foreign 
data  structure  to  a  local  one. 

5 

There  is  sufficient  Control  between 
technologies  necessary  to  establish, 
manage,  and  terminate  the 
integration. 

IRL  5  simply  denotes  the  ability  of  one  or  more  of  the  integrating 
technologies  to  control  the  integration  itself;  this  includes 
establishing,  maintaining,  and  terminating. 

4 

There  is  sufficient  detail  in  the 

Quality  and  Assurance  of  the 
integration  between  technologies. 

Many  technology-integration  failures  never  progress  past  IRL  3, 
due  to  the  assumption  that  if  two  technologies  can  exchange 
information  successfully,  then  they  are  fully  integrated.  IRL  4 
goes  beyond  simple  data  exchange  and  requires  that  the  data 
sent  is  the  data  received  and  there  exists  a  mechanism  for 
checking  it. 

3 

There  is  Compatibility  (i.e. ,  common 
language)  between  technologies  to 
orderly  and  efficiently  integrate  and 
interact. 

IRL  3  represents  the  minimum  required  level  to  provide 
successful  integration.  This  means  that  the  two  technologies  are 
able  to  not  only  influence  each  other,  but  also  to  communicate 
interpretable  data.  IRL  3  represents  the  first  tangible  step  in  the 
maturity  process. 

2 

There  is  some  level  of  specificity  to 
characterize  the  Interaction  (i.e., 
ability  to  influence)  between 
technologies  through  their  interface. 

Once  a  medium  has  been  defined,  a  “signaling”  method  must  be 
selected  such  that  two  integrating  technologies  are  able  to 
influence  each  other  over  that  medium.  Since  IRL  2  represents 
the  ability  of  two  technologies  to  influence  each  other  over  a 
given  medium,  this  represents  integration  proof-of-concept. 

1 

An  Interface  between  technologies 
has  been  identified  with  sufficient 
detail  to  allow  characterization  of  the 
relationship. 

This  is  the  lowest  level  of  integration  readiness  and  describes  the 
selection  of  a  medium  for  integration. 

3.2.  System  Readiness  Level 

The  introduction  of  an  IRL  to  the  assessment  process  not  only  provides  a 
check  as  to  where  the  technology  is  on  an  integration  readiness  scale  but  also 
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presents  a  direction  for  improving  integration  with  other  technologies.  Just  as  a  TRL 
has  been  used  to  assess  the  risk  associated  with  developing  technologies,  an  IRL  is 
designed  to  assess  the  risk  associated  with  integrating  these  technologies.  Now  that 
both  the  technologies  and  integration  elements  can  be  assessed  and  mapped  along 
a  numerical  scale,  the  next  challenge  is  to  develop  a  metric  that  can  assess  the 
maturity  of  the  entire  system  that  is  under  development.  Sauser,  Ramirez-Marquez, 
Henry  and  DiMarzio  (2008)  were  able  to  demonstrate  how  the  TRLs  and  IRLs  for 
any  system  under  development  can  yield  a  measure  of  system  maturity  called  a 
System  Readiness  Level  (SRL).  The  rationale  behind  the  SRL  developed  by  Sauser 
et  al.  (2008)  is  that  in  the  development  lifecycle,  one  would  be  interested  in 
addressing  the  following  considerations: 

■  Quantifying  how  a  specific  technology  is  being  integrated  with  every 
other  technology  to  develop  the  system. 

■  Providing  a  system-wide  measurement  of  readiness. 

The  computational  approach  for  the  SRL  has  been  considered  as  a 
normalized  matrix  of  pair-wise  comparisons  of  the  TRLs  and  IRLs.  The  SRL  matrix 
consists  of  one  element  for  each  of  the  constituent  technologies  and,  from  an 
integration  perspective,  quantifies  the  readiness  level  of  a  specific  technology  with 
respect  to  every  other  technology  in  the  system.  It  should  be  mentioned  that 
although  the  original  (1,9)  scale  for  both  the  TRL  and  IRL  can  be  used,  the  use  of 
normalized  values  allows  for  a  more  accurate  assessment  when  comparing  the  use 
of  competing  technologies.  Thus,  the  values  used  in  the  matrices  [TRL]  and  [IRL] 
are  normalized  (0,1)  from  the  original  (1,9)  levels  by  dividing  each  element  by  9. 

In  addition,  when  no  integration  is  present  between  two  technologies,  an  IRL 
value  of  0  is  assigned.  This  is  in  contrast  to  using  a  value  of  9  when  no  integration  is 
present,  as  was  originally  proposed  by  Sauser  et  al.  (2008).  Using  the  higher  value 
of  9  gave  excessive  weight  to  the  IRL  and  was  distorting  the  overall  SRL  value 
upwards.  Consequently,  this  means  that  in  the  future,  if  the  architecture  is  changed 
such  that  those  two  technologies  become  integrated,  one  can  go  back  and  apply  the 
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corresponding  IRL  value  of  that  new  integration  link.  For  integrations  to  itself,  a  non- 
normalized  IRL  value  of  9  or  normalized  value  of  1  is  used.  The  reason  for  this  has 
a  philosophical  underpinning.  In  the  view  of  one’s  self,  it  is  a  matter  of  a  person 
integrating  various  parts  of  their  personality  into  a  harmonious,  intact  whole  with  the 
purpose  of  keeping  the  self  intact  and  uncorrupted.  For  this  reason,  when 
interpreting  the  integration  of  a  technology  to  itself,  we  define  it  as  uncorrupted  (i.e., 
fully  mature).  If  we  were  to  consider  the  integrations  within  the  technology 
independent  of  the  other  technologies,  then  we  would  be  calculating  a  different  SRL 
and,  thus,  be  considering  a  different  system  independent  of  the  system  of  interest. 


3.3.  Calculating  the  SRL 

The  computation  of  the  SRL  is  a  function  of  the  TRL  and  IRL  matrices: 


Matrix  TRL  provides  a  blueprint  of  the  state  of  the  system  with  respect 
to  the  readiness  of  its  technologies.  TRL,  defined  as  a  vector  with  n 
entries,  is  defined  in  Equation  1 ,  where  TRL/  is  the  TRL  of  technology  /. 


(1) 


[TRL  L 


TRLl 
TRL 2 


TRL 


n 


Matrix  IRL  illustrates  how  the  different  technologies  are  integrated  with 
each  other  from  a  system  perspective.  For  a  system  with  n 
technologies,  [IRL]  is  defined  in  Equation  2,  where  IRL//  is  the  IRL 
between  technologies  /  and  j.  The  hypothetical  integration  of  a 
technology  /  to  itself  is  denoted  by  IRLn. 


(2) 


IRLn 

IRLn  . 

..  IRLU 

P«T„  = 

IRLn 

IRL22 

..  IRL2n 

JRL,a 

iRL„2  ■ 

••  IRLnn- 

In  these  matrices,  the  standard  TRL  and  IRL  levels  corresponding  to  values 
from  1  through  9  should  be  normalized.  A  normalized  value  of  1  for  element  IRL/, 
can  be  understood  as  one  of  the  following  with  respect  to  the  /th  and  y'th 
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technologies:  1 )  they  are  completely  compatible  within  the  total  system;  2)  they  do 
not  interfere  with  each  other’s  functions;  3)  they  require  no  modification  of  the 
individual  technologies;  and  4)  they  require  no  further  integration  linkage 
development. 

In  any  system,  each  of  the  constituent  technologies  is  connected  to  a 
minimum  of  one  other  technology  through  a  bi-directional  integration.  The  way  each 
technology  is  integrated  with  other  technologies  is  used  to  formulate  an  equation  for 
calculating  SRL.  This  SRL  equation  consists  of  the  TRL  and  IRL  values  of  the 
technologies  and  the  interactions  that  form  the  system.  In  order  to  calculate  a  value 
of  the  SRL  from  the  TRL  and  IRL  values,  we  propose  a  normalized  matrix  of  pair¬ 
wise  comparison  of  the  TRL  and  IRL  values. 

Based  on  these  two  matrices,  an  SRL  matrix  is  acquired  by  obtaining  the 
product  of  the  TRL  and  IRL  matrices,  as  shown  in  Equation  3. 

(3)  [®£L,,  =  pR£]„.  x  [™L, 

The  SRL  matrix  consists  of  one  element  for  each  of  the  constituent 
technologies  and,  from  an  integration  perspective,  quantifies  the  readiness  level  of  a 
specific  technology  with  respect  to  every  other  technology  in  the  system  while  also 
accounting  for  the  development  state  of  each  technology  through  the  TRL. 
Mathematically,  for  a  system  with  n  technologies,  [SRL]  is  as  shown  in  Equation  4. 


SRL , 

IRLnTRLx  4-  IRLx^TR1/2  4"  • 

.  +  IRLlnTRLn 

[SRL]  = 

srl2 

= 

IRL,2\TRLx  +  IRL  22  TRL  ^  + . 

.+  IRL2nTRLn 

SRL„_ 

IRL  ]TRL+IRLSRL,  +  . 

i—  n  1  1  n  I  1 

..  +  IRLnnTRLn 

IRLij=IRLj, 


where 


The  representation  of  each  of  the  SRL  values  obtained  in  Equation  4 
addresses  the  first  consideration  previously  discussed  in  Section  3.2.  Note  that 
these  values  would  fall  within  the  interval  (0,n);  so,  for  consistency,  for  each 
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technology,  say  /',  its  corresponding  SRL,-  is  divided  by  ni  (ni  being  the  number  of 
integrations  of  technology  /',  with  every  other  technology  as  dictated  by  the  system 
architecture — including  its  integration  to  itself)  to  obtain  its  normalized  value  between 
(0,1).  The  SRL  for  the  complete  system  is  the  average  of  all  such  normalized  SRL 
values,  as  shown  in  Equation  5.  Equal  weights  are  given  to  each  technology,  since 
they  are  each  identified  as  critical  technology  elements;  in  this  way,  a  simple 
average  is  estimated.  A  standard  deviation  can  also  be  calculated  to  indicate  the 
variation  in  the  system  maturity  and  parity  in  subsystem  development. 

SRL,  SRL ,  SRL,, 

(5)  SRL  =  — ^ ^ - l^- 

n 

where  n,  is  the  number  of  integrations  with  technology  /  plus  its  integration  to 

itself. 

The  SRL  metric  can  be  used  to  determine  the  maturity  of  a  system  and  its 
status  within  a  developmental  lifecycle.  Table  2  presents  an  example  of  how  the 
various  levels  of  the  SRL  scale  can  correlate  to  an  acquisition  lifecycle  (DoD, 

2005a).  The  ranges  of  SRL  represented  in  Table  2  are  derived  from  sensitivity 
analysis  with  sample  systems.  While  we  are  working  to  verify  and  validate  this 
correlation  as  part  of  current  research,  we  contend  that  any  correlation  should  be 
accessed  based  unique  organizational  and  system  development  environments.  Also, 
it  is  important  to  note  that  in  this  correlation,  a  system  that  has  not  reached  full 
maturity  is  capable  of  transitioning  into  a  Production  phase.  This  is  predicated  on 
the  reasoning  that  most  systems  are  deployed  without  all  of  the  technologies  and 
integrations  having  reached  full  maturity.  For  example,  many  military  and  space 
systems  cannot  be  verified  in  their  operational  environment  until  deployed;  likewise, 
many  systems  are  part  of  an  evolutionary  lifecycle  in  which  the  final  maturity  will  be 
verified  once  deployed  or  in  the  next  evolution. 
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Table  2.  System  Readiness  Levels 


SRL 

Acquisition  Phase 

Definitions 

0.90  to  1.00 

Operations  &  Support 

Execute  a  support  program  that  meets  operational 
support  performance  requirements  and  sustains  the 
system  in  the  most  cost-effective  manner  over  its  total 
lifecycle. 

0.80  to  0.89 

Production 

Achieve  operational  capability  that  satisfies  mission 
needs. 

0.60  to  0.79 

System  Development  & 
Demonstration 

Develop  system  capability  or  (increments  thereof); 
reduce  integration  and  manufacturing  risk;  ensure 
operational  supportability;  reduce  logistics  footprint; 
implement  human  systems  integration;  design  for 
production;  ensure  affordability  and  protection  of  critical 
program  information;  and  demonstrate  system 
integration,  interoperability,  safety  and  utility. 

0.40  to  0.59 

Technology  Development 

Reduce  technology  risks  and  determine  appropriate  set 
of  technologies  to  integrate  into  a  full  system. 

0.10  to  0.39 

Concept  Refinement 

Refine  initial  concept;  develop  system/technology 
strategy. 

NOTE:  These  ranges  have  been  derived  from  sensitivity  analysis  with  sample  systems.  They 
are  currently  undergoing  field  verification  and  validation  under  Naval  Postgraduate  School 
Contract  #  N00244-08-0005. 
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4.  Example  of  SRL  Calculation 


To  show  the  steps  and  analysis  involved  in  formulating  the  SRL,  the  following 
example  will  use  notional  data  (with  TRLs  that  range  from  a  low  of  6  to  a  high  of  9 
and  IRLs  ranging  from  5  to  9)  from  a  system  currently  under  development  for  a 
family  of  surface  ships  in  the  US  Navy.  The  system  architecture  analyzed  (see 
Figure  1)  represents  an  end-to-end  integration  of  command-and-control  capabilities 
with  a  variety  of  unmanned  vehicles  and  intelligence,  surveillance,  and 
reconnaissance  sensor  packages.  These  elements  are  capable  of  autonomous 
operations  and  include  both  off-the-shelf  equipment  and  cutting-edge  new 
development  networked  seamlessly  together  to  enhance  effectiveness  and 
efficiency.  For  this  system,  the  following  matrices  can  be  created  for  the  TRL  and 
IRL  (Equations  1  and  2). 
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V 


Figure  1.  Schematic  Architecture  of  System  X 
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0 

0 

0 

0 

0 

0 

0  0  0  0 
0  0  0  0 
0  0  0  0 
5  0  5  6 
0  0  0  0 
0  0  0  0 
0  0  0  0 
0  0  0  0 
0  0  0  0 
0  0  0  0 
5  0  0  0 
9  0  0  0 
9  0  0  0 
9  0  0  0 
8  0  0  0 
0  8  0  0 
9  0  9  0 
0  9  0  7 
9  0  9  0 
0  7  0  9 


As  indicated  in  the  above  integration  matrix,  we  assign  an  IRL  value  of  0 
when  there  is  no  integration  link  contemplated  between  any  two  technologies.  For 
integration  to  itself,  an  IRL  value  of  9  is  used.  Next,  we  normalize  the  [TRL]  and  [IRL] 
matrices  by  dividing  each  element  by  9.  Then,  we  calculate  [SRL]  as  follows 
(Equation  3  and  4): 


(3  and  4) 


M=M2ox 20M: 


SRL, 

SRL 


SRL0 


Table  3  indicates  the  calculated  values  for  each  SRL,. 
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Table  3.  Individual  SRL  Values 


SRL! 

srl2 

srl3 

srl4 

SRL5 

srl6 

srl7 

SRL8 

srl9 

SRLio 

(0,  n,) 

2.000 

3.691 

2.605 

4.482 

1.963 

3.728 

2.000 

2.333 

2.000 

1.519 

(0,1) 

1.000 

0.923 

0.868 

0.640 

0.654 

0.746 

1.000 

0.778 

0.667 

0.759 

SRLn 

srl12 

srl13 

srl14 

srl15 

srl16 

srl17 

srl18 

srl19 

srl20 

(0,  n,) 

1.741 

1.556 

1.444 

1.333 

1.482 

1.568 

5.778 

2.358 

2.099 

2.210 

(0,1) 

0.580 

0.778 

0.722 

0.667 

0.741 

0.784 

0.722 

0.786 

0.699 

0.737 

The  calculated  Composite  SRL  scale  (Equation  5)  of  0.76  indicates  that  the 
system  under  development  should  be  in  the  System  Development  and 
Demonstration  phase  (also  see  Figure  2). 


(5) 


SRL  SRL ,  SRL 

- L  + ^+...+ *■ 

Composite  SRL  =  — — - — - 

n 


SRL ,  SRL,  SRL,n 

2  4 _ 3 

20 


0.76 


Aside  from  the  SRL  providing  an  assessment  of  overall  system  development, 
it  can  also  be  a  guide  in  prioritizing  potential  areas  that  require  further  development. 
That  is,  if  we  are  considering  a  “systems-focused  approach”  to  our  methodology, 
then  we  cannot  evaluate  a  system  based  on  just  a  single  number,  such  as  the 
Composite  SRL.  As  shown  in  our  example  and  illustrated  by  Figure  2,  the  SRL,s 
(technologies  with  their  integration  links  considered)  present  a  spectrum  showing 
some  subsystems  whose  readiness  levels  (i.e.,  SRL/)  are  in  the  three  development 
phases  other  than  the  Composite  SRL’s  System  Development  and  Demonstration 
phase.  While  it  could  be  argued  that  the  overall  SRL  is  only  as  good  as  the  lowest 
SRLi,  this  perspective  would  also  lose  sight  of  even  those  technologies  that  are 
potentially  developing  faster  than  the  system  (see  SRLi, 2,3, 7).  In  understanding  the 
value  of  the  SRL  analysis,  we  must  understand  the  spectrum  of  SRL,  and  its 
relationship  to  the  Composite  SRL  (see  Figures  2  and  4).  For  example,  the  value  of 
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considering  the  IRL  with  the  TRL  is  seen  in  Technology  11.  This  technology  has  a 
TRL  of  9.  However,  when  we  consider  its  IRLs  (both  of  which  are  only  in  level  5), 
we  can  determine  it  not  only  is  less  mature  but  is  a  phase  behind  the  Composite 
SRL.  This  means  that  this  subsystem  (SRL-n)  is  still  in  the  Technology  Development 
phase,  while  the  overall  system  is  already  in  the  System  Development  and 
Demonstration  phase.  In  addition,  as  shown  in  Figure  2,  20%  of  the  technologies 
are  at  least  one  phase  ahead. 

Ideally,  this  type  of  analysis  can  facilitate  strategic  decisions  about 
incremental  technology  and  integration  investments  of  limited  resources.  For 
example,  in  the  upcoming  budgetary  period  or  fiscal  year,  resources  may  be  shifted 
in  favor  of  accelerating  the  development  of  the  technologies  and  integration  links 
that  are  behind  and  temporarily  away  from  those  that  are  ahead — provided  such  a 
shift  is  technologically  and  organizationally  feasible.  This  capability  can  become 
important  when  a  specific  technology  is  a  conduit  for  downstream  technologies — its 
maturity  is  critical  for  the  system  to  reach  a  certain  level  of  maturity.  For  example, 
the  system  diagram  in  Figure  1  shows  that  Technology  4  is  such  a  technology.  If  the 
systems  engineer  has  specified  that  at  this  particular  time  period,  the  SRL  for  this 
subsystem  must  be  at  least  0.80  before  the  rest  of  the  technologies  can  be 
developed  further,  the  program  manager  will  know  that  the  TRL  and  IRL  for 
Technology  4  have  to  be  improved  to  raise  its  SRL  from  the  current  value  of  0.64. 
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SRL 

0.10  0.39/0.40  0.59/0.60  0.79/0.80  0.89/0.90  1.00 


5%  75%  5%  15% 

- 1 - 

20% 


Figure  2.  SRL  Mapping  to  Defense  Acquisition 
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5.  SRL  Relevance  and  Future  Research 


Given  the  ability  to  estimate  readiness  of  a  system  under  development 
(summarized  in  Figure  3),  organizations  can  systematically  evaluate  the  implications 
of  using  alternative  technologies  or  system  architectures,  prepare  development 
plans  that  optimize  the  objectives  of  the  development  team,  and  eventually  be  able 
to  evaluate  and  monitor  the  progress  of  the  development  effort  to  identify  problem 
areas  and  corrective  measures  (example  in  Figure  4). 


Figure  3.  SRL  Methodology  and  Analysis  Flow 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 


-23- 


NOTE:  ALL  DATA  IN  THIS  TEMPLATE  IS  NOTIONAL 


Platform  2 
Platform  1 


Compos*®  Actual 

Planned 

SRL 

.57 

.6 

53 

.6 

LEGEND 

|  Platform  1  System 
Platform  2  System 
Current  Composite  SRL  Status 
Previous  Composite  SRL  Statu* 
Individual  technology  SRL  Status 
Q  Technology  Readiness  Level 
Q  Integration  Readiness  Level 
Q  System  Readiness  Level  Demarcation 
”  *"  Scheduled  Position 

Low  Risk  to  Cost  and/or  Schedule 


V 

V 


Moderate  Risk  to  Cost  and/or  Schedule 
High  Risk  to  Cost  and/or  Schedule 


Date  Collection  Period:  XXSXXKX  -  X/XX/XX 
Previous  Report  Date:  XXOUOXX 
Schedule  Updated  OV2SA7  (OCR) 

Tech  12  Tec 


SRL 


Tech  i  Tech  6  Tech  10  Tech  T  Tech  »  .  Tech  12  Tech  11  Tech  9  Tech  4 

,  ¥  O  ,  o  o 


Figure  4.  Example  of  Documented  Status  via  Roll-up  Chart 

In  our  development  of  an  SRL  scale,  we  strived  to  maintain  a  systems- 
focused  approach  that  would  create  a  metric(s)  to  address  some  of  the  current 
concerns  with  the  TRL.  What  resulted  was  a  set  of  metrics  and  an  approach  that 
can  have  the  following  implications  on  defense  acquisition: 

■  The  SRL,  IRL,  and  TRL  provide  an  enhanced  capability  alignment 
through  the  identification  of  specific  technology,  integration,  and 
system  maturities  that  can  be  used  as  a  trade-study  tool  to  select  the 
most  appropriate  technologies  and  integrations  to  obtain  the  lowest 
amount  of  risk,  cost,  and  time  and  satisfy  a  given  customer  need. 

■  The  SRL  [IRL,  TRL]  model  can  improve  customer  confidence  in  the 
acquisition  manager  by  providing  a  qualification  of  system  maturity  in 
relation  to  system  functionality.  It  can  also  provide  improved 
understanding  of  the  system’s  mission  capabilities  in  terms  of 
readiness  criteria. 

■  The  SRL  can  provide  an  assessment  of  maturity  at  multiple 
architectural  layers.  Any  single  SRL  assessment  contains  multiple 
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SRL  assessments  from  the  SRL  vector,  which  can  provide  insight  into 
the  interdependencies  of  different  sub-functions  and  how  they  fit  within 
the  larger  architecture. 

■  The  SRL  can  provide  a  fast,  iterative  assessment  that  can  be  repeated 
and  traced  during  development.  This  can  facilitate  a  valuable  exercise 
in  architecture  examination  and  creation,  which  can  allow  for  better 
system  understanding  and  (re)formation. 

■  The  SRL  and  IRL  allow  for  other  factors  (in  addition  to  technology 
readiness)  as  measures  of  maturity.  In  addition,  decision-makers  can 
consider  factors  such  as  obsolescing — by  comparative  analysis  of 
multiple  technologies  to  acquisition — and  the  optimization  of 
technology  maturation  investment  and  transition  funding.  This  is 
currently  an  area  of  future  research. 

■  The  SRL,  IRL,  and  TRL  provide  common  ontology  to  measure  and 
describe  acquisition  development,  system  development  and 
technology-insertion  evaluation. 

■  The  IRL  reduces  the  uncertainty  involved  in  integrating  a  technology 
into  a  system  and  identifies  integration  as  a  separate,  specific  metric. 

Despite  the  utility  of  the  SRL,  it  is  not  without  a  core  limitation.  That  is,  our 
tactical  approach  to  the  SRL  was  similar  to  that  of  calculating  a  student’s  grade  point 
average  (GPA) — in  which  ordinal  data  is  given  numeric  value  in  order  to  assess 
overall  progression  or  performance.  This  approach  also  incurs  a  key  limitation  to 
assessing  a  system’s  development.  Accordingly,  the  SRL  for  one  system  cannot  be 
compared  to  the  SRL  of  another  system  unless  they  are  the  same  system.  For 
example,  it  is  difficult  to  compare  a  student  with  a  3.2  GPA  (on  a  4.0  scale)  in 
physics  with  a  student  that  has  a  3.8  GPA  in  biology.  These  students  belong  to 
different  systems  of  education,  but  they  are  evaluated  with  the  same  system  of 
metrics.  Likewise,  the  SRL  can  be  effective  for  assessing  the  progressive  maturity 
of  the  system  of  interest,  but  it  is  questionable  to  compare  the  maturity  progression 
of  two  systems  against  each  other  because  of  other  inherent  factors  related  to  the 
context  in  which  the  system  is  being  developed. 

Further  trials  using  real  case  studies  are  necessary  in  order  to  verify  the 
formulation  of  the  SRL,  as  well  as  to  establish  its  validity.  These  will  also  be 
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necessary  in  order  to  illustrate  the  benefits  of  SRL  in  terms  of  improved  risk 
management  and  value  added  at  key  decision  points  along  the  acquisition  lifecycle. 
When  the  validity  of  the  SRL  is  established,  it  can  then  be  expanded  to  incorporate, 
where  necessary,  other  measures  of  readiness,  such  as  Manufacturing  Readiness 
Level  (MRL).  As  with  any  research,  the  fundamental  objective  is  to  increase  our 
understanding  by  asking  questions  that  lead  to  more  questions.  Thus,  for  future 
research  in  system  maturity  assessment  and  defense  acquisition,  we  propose  some 
of  the  following  questions: 

■  Are  there  variations  in  how  system  maturity  assessment  is  used  with 
various  lifecycles,  e.g.,  linear  acquisition,  evolutionary  acquisition, 
revolutionary  acquisition? 

■  What  are  the  implications  of  system  maturity  levels  for  the  integration 
of  open  systems  into  evolutionary  acquisition? 

■  What  are  the  impacts  of  disruptive  technologies  on  systems  maturity 
forecasting? 

■  How  does  vendor  selection  impact  system  maturity  assessment? 

■  How  do  other  maturity  metrics,  such  as  the  Manufacturing  Readiness 
Level  (MRL),  work  with  the  IRL  and  SRL? 

■  How  can  the  techniques  of  system  maturity  assessment  be  used  for 
trade-off  analysis  of  competing  technologies  or  systems? 

■  What  are  the  impacts  of  obsolescence  to  system  maturity  planning  and 
road  mapping? 

■  What  are  the  single-technology  refreshment  optimization 
considerations  for  asynchronous  refreshment  frequency? 

■  What  are  the  multi-objective  optimization  considerations  for 
asynchronous  refreshment  frequency? 

■  What  are  the  uncertainties  surrounding  the  lifecycle  curve  for  system 
maturity? 

■  How  can  we  consider  the  environmental  costs  throughout  a  system’s 
lifecycle? 
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6.  Conclusions 


We  contend  that  the  IRL  is  necessary  because  in  some  programs,  integration 
elements  have  been  overlooked  and  have  resulted  in  major  debacles.  We  also 
introduced  the  development  of  a  system-focused  approach  for  managing  system 
development  and  for  making  effective  and  efficient  decisions  during  the  defense 
acquisition  process.  To  accomplish  this,  we  developed  a  SRL  scale  incorporating 
both  the  current  TRL  and  the  proposed  IRL  scale.  We  then  described  the 
foundations  of  the  SRL  and  demonstrated  the  techniques  for  determining  current 
readiness  of  a  system  to  determine  its  position  in  the  defense  acquisition  lifecycle. 
We  summarized  our  approach  (describing  how  it  may  be  used  within  defense 
acquisition),  showed  a  specific  example  of  how  the  analysis  could  be  reported,  and 
provided  some  questions  for  future  research. 

The  DoD  Technology  Readiness  Assessment  (TRA)  states  that  “the  TRA 
should  not  be  the  sole  means  of  discovering  technology  risk”  (DoD,  2005b). 
Furthermore,  as  stated  earlier,  the  GAO  has  reported  that  the  DoD  needs  additional 
metrics  for  evaluating  weapons  systems.  While  metrics  can  identify  critical 
parameters,  establish  milestones  to  assess  progress,  provide  direction  for  risk 
management/mitigation,  or  sustain  entry  and  exit  criteria  for  major  milestones,  we 
must  keep  in  mind  the  four  guidelines  for  effective  and  efficient  metrics  by  Dowling 
and  Pardoe  (2005)  as  described  earlier.  Accordingly,  we  attempted  to  follow  these 
guidelines  and  proposed  the  inclusion  of  a  separate  maturity  scale  to  measure  the 
progress  of  the  development  of  the  integration  links  of  a  system  and  the  system  as  a 
whole. 


We  consider  the  TRL  to  be  simple  and  understandable;  however,  some 
ambiguity  exists,  in  part  due  to  the  extrapolation  of  the  TRL  beyond  what  it  was 
intended  to  do.  We  believe  that  the  IRL  mimics  the  value  of  the  TRL  in  that  it  is 
simple  and  understandable,  but  we  contend  that  the  interpretation  of  the  individual 
IRL  levels  may  need  more  clarification  before  the  IRL  can  become  a  metric  in 
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practice.  The  combination  of  the  TRL  and  IRL  for  the  formulation  of  the  SRL  was 
not  a  simple  endeavor,  as  many  alternative  mathematical  approaches  were  pursued 
(Sauser  et  al.,  2007).  The  chosen  approach  was  used  because  it  was  the  simplest 
and  most  robust  with  respects  to  its  sensitivity  to  changes  in  any  TRL  or  IRL  within  a 
system.  While  the  addition  of  any  metric  means  incurring  additional  costs  for  an 
organization,  we  consider  the  addition  of  the  IRL  and  SRL  as  a  cost  savings,  as  they 
are  able  to  identify  factors  that  have  been  significant  failures  in  many  system- 
development  programs.  Finally,  we  attempt  to  focus  the  development  of  these 
metrics  based  on  data  that  would  normally  be  available  to  any  systems  engineer 
(e.g.,  system  architectures,  baselines).  Even  with  what  we  consider  to  be  a  valuable 
contribution  to  the  assessment  of  system  maturity,  the  additive  value  of  “readiness” 
metrics  carries  with  it  the  additive  drawbacks:  (a)  Subjectivity  and  Human¬ 
intensiveness — human-intensive  assessments  can  be  overly  optimistic  and  contain 
inherent  variation  or  ambiguity  that  is  averaged  away  and  which  some  of  the  existing 
approaches  may  fail  to  prevent;  and  (b)  Limited  Focus — while  this  is  not  the  intent, 
focusing  on  single  or  a  limited  subset  of  numbers  can  draw  attention  away  from 
other  core  issues. 

In  conclusion,  the  conceptual  development  of  these  or  any  metrics  and  tools 
have  outpaced  their  validation  and  verification  in  the  field.  What  is  necessary  now  is 
to  have  greater  involvement  from  practitioners  so  the  acquisition  community  can 
agree  to  a  common  measurement  and  language  that  can  only  improve  the  system 
development  and  acquisition  process. 
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